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ABSTRACT 

Based on Leviton and Hughes' conceptualization of 
variable Clusters that affect evaluation utilization, a procedure for 
measuring utilization potential was developed. Five clusters of 
variables -ate consistently related to utilization: relevance, 
credibility, communication, information processing, and user 
involvement and advocacy. The communication 1 and information 
processing factors were combined for this study. To relate the four 
remaining factors to decision-making within the context of Title I 
programs, Title I decision area? were identified. They include fund 
allocation, program adoption or change, Staffing, student selection, 
and test selection. The generation of items for the pilot instrument 
was a multistage process involving Title I Technical Assistance 
Centers across the country** From the several hundred items generated, 
a 65-item pilot instrument was developed an<^ pilot tested. Maximum 
likelihood factor, analysis using a subset of items revealed a set of 
correlated factors related to Levitcm and Hughes' conceptualization. 
The exploratory factor analysis results were not confirmed on a 
second sample, but this may have bee^due to small sample sizes. This 
research indicates that the possibility of developing a scale to • 
assess utilization potential is very real. (Author/BW) 
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^ I Introduction 

During the 1970s there was considerable focus on evaluation utili- 
zation and many articles began to appear on this topic. More 'recently, 
several analytic reviews have appeared (e.g., Haenn, 1980; Hansen, Martin 
and Oxford, 1979; Leviton and Hughes, 1981). The reviews all attempted 
to identify clusters or "utilization factors" and involved at least two 
and usually three or more levels of factors with each major .factor 
divided into subfactors. 

The three reviews mentioned here had a fair amount of overlap in 
terms of references, but approached the task through somewhat different 
perspectives. The review by Haenn was considered in the context of 
school district utilization; the Hansen et. al. review was considered in 
the context of Title I evaluation. The Leviton and Hughes review con- 
sidered utilization in a much broader context, not restricted to school 
related environments. Additionally, the Leviton and Hughes paper devoted 
considerable attention to the definitional and methodological problems 
inherent in research on evaluation utilization. * 

Much of the research on utilisation has, of necessity, relied on the 
use of interviews (e.g., Caplan, 1977; David, 1978) and intensive case 
studies (e.g. , Alkin, et . al\ , 1979; Pat ton, 1978) and questionnaires. 
Leviton and Hughes cite four major problems these techniques have with 
respect to utilization research: 1) "•••it is difficult to document that 
utilization occurs, because evaluations are frequently used informally. .. , 
2) "demonstrating that change, at any level, was caused at least in part 
by evaluations:. . . ," 3)". . .the question of base rates .for comparison. 
Until, recently, we believed the base rate for utilization wks very low. 
We are learning that the fault may lie with our measures. •• ," and 4) 



".♦•the unit of analysis: What is an instance of utilization? It is 
necessary to quantify utilization if we are to show that it can be 
enhanced. 11 (Leviton and Hughes, p. 533). Although not a methodological 
problem per se , one might also add cost considerations as a problem with 
interview and case study approaches to research on utilization. 

Given the methodological problems inherent in much of the utiliza- 
tion research, one might ask the question: Is there any way of develop- 
ing a scale that would measure' utilization directly? The answer is 
probably in the negative, but it might be possible to develop a scale 
that would measure "potential" for utilization where the higher the 
potential the more likely it is that utilization occurs. Working within 
the utilization conceptual framework developed by Leviton and Hughes, 
this paper reports preliminary results of a pilot study designed to 
develop such an instrument. Briefly described in the next section are 
the five clusters (hypothetical factors) conceptualized b^Leviton and 
, Hughes. 

Hypothetical Factors 

iton and Hughes identified five clusters of variables that seemed 
consistentlyffe^ated to utilization. The five clusters are: 1) Relevance, 
2) Credibility, 3p Communication, 4) Information Processing and 5) User 
Involvement, an<T Advocacy. - Each of these is briefly discussed below. 

Relevance . The primary concern of this category is whether the eva^ 
uatiori meets Che user's needs. This is viewed as essential to utilization. 

Relevance assumes that the evaluation answers the necessary questions or 

* * 

measures goals that are viewed as important by users. One variable within 
thif cluster is timeliness: the need to have evaluation results in time 



for consideration before key decisions are made. 

i • • 

» Credibility . Credibility of the information and the information 
producer also affects utilization. Credibility* of evaluation as it is 
compared to alternative available sources of .incarnation, preconceptions 
or biases of users toward research, trust in the research, and perceived 

quality of the information are some aspects of credibility that affect 

v 

utilization. 



Communication * Another category of variables is broadly referred 
to as communication. The pertinent variables included are associated 
with communication as it occurs within the bureaucratic hierarchies where 
the evaluation is focused. Important to utilization is the frequency of 
contact between the producers of the evaluation ajid the potential users. 
Also important is the existence of networks in the bureaucratic hierarchy 
which assure that, during dissemination, valuable information is not left 
out or distorted. 

Information Processing * Information processing refers to the impor- 
tance of^translating evaluation findipgs into specific implications* Eval- 
uations must be structured to answer specific questions, and they should 
be clear in terms of their goals and objectives. The information process- 
ing style of the administrator (user) , generally different from the 
^valuator's style, has implications for utilization. Evaluations should 
be presented in a style familiar to the user. * 

I 

* User Involvement and Advocacy . User involvement and advocacy are 
variables of a political nature* Utilization is affected by Lhe level . 



of interest and commitment of decision makers to the process of evalua- 
tion as well as ,to the program being evaluated. The direction of the 
evaluation results, i.e. , whether or not they support the | advocate's 

position, will impinge upon utilization, 

» 

Four Factors . For our purposes, it was felt that the "Information 

\ 

Processing" cluster or factor had considerable overlap with the "Communi- 
cation" factor and we elected to combine the two factors. Our interest 
was in the development of a scale that would assess utilization potential 
within the context of Title I decision-making. Using the conceptual 
framework developed by Levitoa and Hughes, recced to four factors : 1) 

4 

Relevance, 2) Credibility, 3) Communication, and 4*) User Involvement and 
Advocacy we believed such a scale, could be developed. To relate the in- 
strument to decision-making, Title I decision areas were identified as 
discussed in the next section. 

\ 

^Decision Areas > ^ 

Within Title I, at least five areas were identified where school 
district personnel made decisions. These areas were: 1) fund allocation, 
2) program adoption or change, 3) staffing, 4) student selection, and 5) 
test selection. Evaluation has the potential for informing administrators 
making decisions in each of these areas as described below. 

gund Allocation . School administrators are always making decisions 
concerned with funding. In a Title I context, such decisions might be 
concerned with how funds should be allocated to different projects (e.g., 
reading and math projects). 

Progr am Adoption or Chan ge. Decision-making in this area is concerned 

. • i 

*with selection of a program beat meeting the needs of students served or 



with how an operating program might be improved. Which program will 
best serve the needs of students? Would increasing timeNpent in drill 
improve the program? These are example questions that .imply decisions 

1 - 

required in the area of program adoption or change. 

Staffing . Staffing decisions might include deciding what staffing' 
pattern is required, deciding which staff should participate ^or what 
staff training is necessary to successfully Implement or ^operate a 
program. 

Student Selection . Deciding which students should participate in a 

program is an obvious decision that must be made. Perhaps-more crucial 

in this area is deciding what selection process should be implemented. 

Test Selection . Deciding on a testing program that serves multiple 

purposes can involve many decisions. For example, "Does a test accurately 

reflect program goals?" or "Should out-of-rlevel testing be used?" Are 

questions which need to be considered when test s'el'e&tion decisions are 

being made. # 

. Summary of Decision- Areas . The-, cfecision areas briefly discussed in 

this section are only a few of the areas where school district administra- 

* « 
tors make decisions. The five areas—fund allocation, program adoption 

or change, staffing, student selection, and test selection were discussed 

in this section because these areas are general across school districts 
» 

and Title I. ' 



, , Method 

Devel opment of the Instrument 

" K '. ' * 

The generation of items for the pilot instrument was a multistage 
process involving Title I Technical Assistance Centers (TACs) across the^ 
country. Each TAC receiv^^^^ concept paper describing the "factors 11 and 
our approach to developing ltetos, a brief description of the "factors" 
and a matrix showing the five decision areas within which Title I decision- 
making was likely to' occur 6the btief description and matrix are in Appen- 
dix A). Each TAC was asked to develop Items for each decision area 
within two "factors." 

From the several hundred itetns generated, after categorizing and 

editing, a sixty-five item pilot instrument was developed. This instru- 

» * 

ment was reviewed by state and local school district staff and by TAC 
.staff. Based on reviewer comments, additional editing was done before a 
final piJLot instrument was distributed* 

instrument requested respondents to rate each of the sixty- 
five items o\tw£ dimensions—the degree to which the described condition 
existed in the district and the importance of the condition. Both ratings 
were done on five point scales with a "1" indicating little existence (or 
importance) and a "5" indicating considerable existence (or importance). 
Although the instrument was very lengthy, all items were retained for 
pi^ot testing with the intent of reducing the instrument to about thirty 
items in its final form. 

Sampling 4 

| Three states agreed to allow pilot testing of th^ instrument. From" 
lists of school districts having, a Title I pupil enrollment of at least 




7 • * 

* / 

100, random * samples of 100 districts per stSte were drawn.. Each state 
sample was then randomly split into two samples (I and II) of size 50. 
Sample I instruments requested respondent identification (name* business 

address and telephone number) while Sample II instruments did not request 

< 

respondent identification* Both instruments had ^a page requesting back- 
ground information (position, number of years in TitJ.e I, highest degree 
earned, grade levels served by Title I programs, and appriximate number 
of children served «in Title I programs )• A comparison of Sample ^1 and 
Sample II responses on the background information is given in Table 1« A 
total of 223 instruments were returned, 114 from Sample I districts and 
109 from Sample II districts * indicating a slightly higher return rate from 
the "respondent identification" Sample I districts. Across both samples 
\he district median number of pupils served in Title I reading programs 
was 140 and the median number served .in Title I math programs was 75. 

Phase I Analyses 

Each item was classified accbrding to the hypothetical factor for 
which it had been developed — Relevance (R), Credibility (Cr), Communica- 
tion (Co), and User i Invo.lvement (UI).. Reliabilities were computed for 
each hypothetical factor separately for the existence (ES) and Importance 
(IS) scales. Results of these analyses are shown in Table 2^ As shown 
in Table 2, the hypothetical factor reliabilities were quite respectable 
(minimum of .81) » 

Factor analytic techniques were then -applied separately to each 
scale, a "very simple structure" (VSS) analysis was done (Revelle and 
Rocklin, 1979) Results of these analyses * were used to specify para- 
meters for maximum likelihood confirmatory factor analyses using LISREL 
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Table 1 



Background Information from Sample I (Respondents Identified) 
And Sample II (Respondents Not Identified) Questionnaires 



1 


"Question". 


Sample I 
(n«114) 


Sample II 
(n-109) 


Total 
(n=223) 


Position* 

Administrator 
Evaluator 
Teacher 
Other 


■76 
49 
28 
25 


79 
31 

6 


. 155 
80 

31 


\ 

Year s* Experience 
in Title I 

5 yrs or less 

6 yrs - 10 yrs 
more than" 10 yrs 


• 

49 
32 
33 


29 
31 


Qft 

61 
63 


Highest Degree 
Doctorate 
Specialist 

Masters 
Bachelor 


6 
20 

■ ,61 
" : 27 

\ 


7 
22 
56 
24 


13 
42 
117 
51 


Respondents checked all that 


applied. 5 
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Table 2 



Phase I .Hypothetical Factor Scale Reliabilities 







"Hypothetical 


Number 


Reliability 




Factor 


of Items 


ES IS 




Relevance (R) 


15 


.88 .89 




Credibility (Cr) 


11 


.81 .82 




Communication (Co) 


20 


.88 .90 




User Involvement (UI) 


19 


, .91 .91 



10 

^Joreskog and S'drbom, 1978). Both LISREL analyses resulted in significant 

! 8 indicating that the four facto^ model, as specified, was n<& con- 
firmed. These results wer£ somewhat puzzling, VSS analyses indicated one 
large general factor dominated by User Involvement items , but also 
including Relevance and Communication, items. It seemed quite possible 
that the, LISREL models were misspecif ied. 

Since the concept of a scale to measure evaluation utilization poten- 
tial is relatively new, we felt additional , more exploratory, analyses 
should be conducted. These analyses are described in the next section 
on Phase II analyses. 1 

Phase II analyses 

v The main concern of the Phase II analyses was to conduct exploratory 
factor analyses that would lead to a set of factors that could be con- 
firmed in subsequent analyses. To accomplish this, we decided to randomly 
split the samples into two subsamples (Sample A and Sample B)* Sample A 
would be used for exploratory analyses and Sample B would be used for 
confirmatory analyses. This decision created another problem, subsample 
n's would most likely be much* too smalj. to factor analyze the full 
65-item set. It was decided to work with a 30-item subset. \ 
ItenuSelection . to select the 30-i'tem subset, we first had the 65- 

m 

items independently classified into tfre four hypothetical factor cate- 
gories (R, Cr, Co and Ul) by thr.ee raters. A total of 60 items were 
assigned to the same hypothetical factor category by at least two of the 
raters while 27 of the items were assigned to the same category by all 
three raters. Twenty-seven items of the 30-itjem subset were those items 
assigned to a category by all three raters. Since only four User Involve- 



'11 

» 

ment (UI) items were within the 27 items, three additional UI items were 
selected at random from 1 the items, that two raters .placed in the UI 
category. Some items were placed in categories different from their 
original category designation. Most of the 27 (23 )« were cateogrized^ by 
all three raters into their original Category. ^The 30 items are' given 
in Appendix - B. Table 3 'shows reliabilities for the Existence Scale 
hypothetical factors based on the 30-item subset. 

Table 3 

Phase. II Hypothetical Factor Existence Scale 
Reliabilities for Subsample A, Subsample B 
and Total Sample 





Hypothetical 


Number * 


Subsample* 


Total 3 


Factor 


of Items 


A -B 


Sample 






(109) (114) 


(223) 


Relevance (R). 


7 


.80 .82 


.81 


Credibility (Ct) 


- 4> 


.76 .77 


.77 * 


Communication (Co 


) 10 


.82 .83 - 


.82 


Ufcer Involvement 


(UI) 7 


.76 .75 


.75 



a San*pie n's are given in parentheses. 



Subsample A exploratory analyses > Exploratory common factor analyses 
using maximum likelihood procedures developed and described by Joreskog 
and Van Thillo (1971) were .done with data from Subsample A ? . Multiple R 2 
coefficient s . were usred for initial communality estimates. Since it jjas 
assumed that the hypothetical factors were correlated, oblique rotational 
procedures were employed The SPSS Subprogram JFACTOR (Burns, 1977) was 1 

used for the exploratory analyses. 

• • * * 

The JFACTOR program also'prints results of three statistical tests 

■> 

ta determine the suitability of, a correlation matrix for factor analysis 

* • « 

\ 
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(Dziuban and Shirkey, 1974). Bartletts's test of sphericity was rejected 
( ^Xiftf 1239 .05, p<.001) indicating the Stibsample A correlation matrix was 
suitable for^actoring. Inspection of the off-diagonal elements of the 
anti-image matrix showed 13.33 percent of the .elements were greater than 
zero (>. 09) / small enough to indicate the matrix suitable for factoring. 
Finally, Kaiser's measure of sampling adequacy of .78 (almost the "merito- 
rious" range in the .80s) indicating the matrix was suitable for factoring. 
(Dziuban and Shirkey provide a brief discussion of each test.) ■ 

Nine factors were extracted from the Subsample A correlation matrix 
before a nonsignificant Chi Square value was reached. (Av significant Chi 
Square value indicates that a significant amount of variance remains in 
th£ residual correlation matrix.) The pattern matrix was rotated obliquely 
using the Kaiser Normalization procedure. Application of the Scree test 
(Gorsuch, pp. 152-156) and visual inspection of the rotated structure 
matrix indicated that at, least five of the factors were interpretable. 

As a check 'that the ^rotated pattern matrix attained simple structure, 
• #* 

the five criteria by Tburstone (Gorsuch, pp. 164-165) were applied. The 

i 

-results of the tests for simple structure are given , in Table 4. 
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Table 4 



Thurstone.^s Five .Simple Structure Criteria 
Results, on Subsample A Rotated Pattern Matrix 



Criteria 

1* Each variable has at least 
oae zero loading. a 

2. Each factbr has at least one 
set of linearly independent" 
variables with zero factor 
loadings. ' 

3. For every .pair of factors 
there are several variables 
whose loadings are zero on 
one factor but not the other. 



4. For every pair of factors a 
large proportion of varia- 
bles have zero loadings on 

/both whenever Aore than 
about four factors are 
present. 

5. For every pair of factors 
only a small number of 
variables should Skatfe 
nonzero loadings on both 
factors. 



Met 
, Yes 
Yes 



582 of the factor 
pairs had 502 or 
more of the varia- 
bles meet this cri- 
teria. 

972 .of the factor • 
'pairs had 202 or 
more of the varia- 
bles with zero load- 
ings on both 'factors. 



922 of the factor , 
pairs had 202 or f * 
fewer of the varia- 
bles with nonzero 
loadings on both '„ 
factors. 



a A zero loading was defined as any loading strictly leas than .1 in 
absolute value. 



The rotated pattern matrix appeared to meet the criteria for simple struc- 

ture. The Subsample A correlation matrix, rotated factor pattern and 

, structure matrices, and the factor correlation matrix are given in 

Appendix C. ' 

The interpretation of factors found jin Subsample A is based for the 

most part on*the structure matrix f tsee Appendix C). This matrix gives 
» 

the correlations betyeen a variable and a factor. 
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The items in each of the four pre-determined categories (R, Cr, Co and 

DJ) tended to correlate highly with one, and i>n some cases two, of the 

factors resulting from the analysis. Six of the seven User Involvement 

• * 
items correlated highly with the sixth factor of the nine factor configu- 

ration. ( . * 

The Credibility items correlated highly for the most part with fac- 
tors 1 and 3, having somewhat higher correlations on the average with 
factor 3. 

The ten Communication* items correlated most highly with factor 4, 
with a substantial number also correlating (though not as hjlghly) with 
factor 6. 

The fourth category of items, Relevance, seemed to be the most defin- 

ltive m terms of the factor structur£\ These items" clearly and nearly 

L 

exclusively correlated on factor 5. 

Hence, the oblique rotational procedures resulted in items within 
the categories correlating primarily % with a total of five factors, with' 
the remaining four factors being of relatively small significance. 

The next step in the analysis, and often the most difficult one in 
factor analysis, was an attempt to draw an .interpretation of the major 
resulting five factors an light -of the four-item categories. In an 

effort to further interpretation of the factors, the content of each 

I 

item was examined in relation to factors with which the item was highly 
correlated. # 

'The first area for investigation of item content was for those items 
which correlated significantly with more than one factor. For example, 
the five items within the Communication -category which correlated signifi- 
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I ! 
cantly with both factors 4 and 6 were examined oft item content. Four of 

thes'e ifive items haH correlated highly ^with factor 4 which, upon closer 
investigation of Item content, could be interpreted as a "Communication 11 
factor. Of the ten total Communication items, five items correlated 
significantly with factor 6, which appeared to be a "User Involvement" 
factor. The wording of the five items which overlapped factors 4 ,and 6 
(Communication and User Invalvement) was examined. In four of these five 
pre-categorized Communication items the words "used" or "useful" terms 
were found, indicating reason why some sampled respondents t might have 
interpreted such items in a "user invdlvement" sense as well as in a 
"communication" sense. 

The other item category which contained a number of items which cor- 
related on more than one % f actor were the Credibility items — four of six 
correlating highly with both factors 1 and 3. - An examination of the 
terminology for these four items which had correlated with factor 1 
showed the use of "decison-making" terms within the content of each item. 
Since factor 3 had been temporarily termed the "Credibility" factor, it 
seemed worthy to examine the content of items which correlated only with 
factor 1 and no other factors, and those Credibility items which did not 

r 

correlate with factor 1. Othfer items which correlated with factor 1, and 

not with other factors, had similar "decifriQn-making" terminology within 

* r _ 

them. Factor .1, therefore, seemed to. be interpretable aa -a Cecis ion- 
Making factor, different from the four j££e-detjermined categories* Those 

Credibility items which did not correlate with the Decision-Making factor 

") 

1 did not contain terminology mentioning "decisions," but rather could 
be termed "pure" Credibility items., . 
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In summary, fhe exploratory analysis on^-Sub sample A produced a nine-" 
factor structure. Four of these factors were relatively minor in terms 
of the number of items within any of the four pre-determined categories 
which correlated highly on any otle of the factors, i.e. item correlations 
with these four factors were generally smaller and dispersed across all 
four pre-determined categories of items (R, Cr, Co and UI). The^ interpre- 
tation of the exploratory factor analysis is that the four pre-determined 
categories of items correlated differentially primarily on five factors, 
which upon examination of the itemfe with respect to the, factors resulted 
in the factors being interpreted as Relevance, Credibility , |Communication, 
User Involvement and Decision-Making. 

Subsample B confirmatory analyses . The program LISREL (J5reskog and 
S&rbom, 1978) was used for confirmatory analyses. Basically, by using 
the Subsample A pattern matrix and factor correlation matrix (see Appendix 
C) with the Subsample B correlation matrix a maximum likelihood test on 
the *residual matrix is available. The relationship between the matrices 
is shown below. \ 



Where 2> for our case is the correlation matrix determined by the factor 
pattern matrix (^y)> the i^ferffcctor Correlation matrix Cj[ ) and the er- 
ror matrix (^). B is an identity matrix. The maximum likelihood .test 
performed is on §«£ where J is the Subsample B correlation matrix and £ 
is estimated from the above equation. * A nonsignificant "X^indicsLtes the;, 
results from Subsample A were confirmed on Subsample B. 

When estimates 1 for both ^ttfmd wetfe fixed and taken from Subsam- 



ple A results, M most stringent test, the factor pattern and correlations 



17 

were not confirmed ( K>wf 717,41, p<*0000). This test required confirma- 
tion of both the factor pattern and the intercorrelation among factors, 

I 

When dnly estimates for -Ay were fixed, the test, if nonsignificant, would 
conf irtp^tJJhv Subsample A factor pattern, but not the intercorrelation 
among factors A This test also did not confirm the Subsample A pattern 
matrix (^^585.21, p<.0000). Other, less restricted confirmatory analy- 
ses (e.g. , J\,y estimates only partially fixed) are currently being con- 
sidered. 

Discussion, Summary and Implications 
Based on the Leviton and Hughes' conceptualization of variable clus- 
ters that affect utilization, a procedure for measuring utilization 
potential was described. ^Preliminary results of the pilot effort, within 
the context of Title I evaluation utilization, were presented* 

Maximum likelihood factor analyses using a subset of items with a 

I 

random half of the data base .clearly indicated that a set of correlated 
factors related to the Leviton and Hughes' factor conceptualization were 
found. Additionally, a decision-making factor also appeared that may 
have been due to the wording of some items * ^ 

The exploratory factor analysis results were not confirmed on a second 
sample. However, this should not be taken as an indication that either 
the Leviton and Hughes conceptualization was not confirmed or that an 
instrument for measuring utilization potential is not possible. By split- 
ting the sample, our effective n's became quite small for the application 
of factor analytic techniques. When sample size is considered, we feel 
the results are highly suggestive. The possibility of developing a scale 



i 
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to assess utilization potential is very real and its <|evelopment^j^uld 
provide a powerful methodological tool for utilization research and 

for identifying areas where training might increase utilization/ \ 

\ K 
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tilization Factors 



\ 



Leviton and Hughes (1980) have proposed a five factor structure for the construct 
of utilization. Although it is possible to question any or all of the proposed, 
factors , .we believe that our goal of measuring utilization will be more readily 
achieved by using a relatively concise conceptualization of the construct than 
by attempting to employ one of the more elaborate models described^4a' the litera- 
ture • 




As described by Leviton and Hughes, each of the five factors ^bsociated with 
'evaluation utilization is comprised of a cluster of discrete variables ♦ We have 
chosen to paraphrase, and call thes^ variables "facilitating conditions": 



Factor 
1 • Relevance 



a. 



2* * Communication' 



Facilitating -Conditions ; 

Evaluation addresses client needs; ie. in 
a Title I program, evaluation data 
are perceived as relevant to information 
needs \ of teachers, program people, and 
administrators. 

Evaluations provide timely information, 
i.e. data are available when needed for 
decision making. 

There is formal and informal interaction 
between evaluator and users of 
evaluation information (teachers, program 
people, administrators). 



b. As information filters through the bureau- 
cratic hierarchy, no information is distort! 
or omitted. 

c. The evaluation information communicated to 
each user is specific to his/her needs ♦ 

d. The information communicated to each user 
is in a form he/she can readily ' assimilate* 



3. Credibility 



Information from evaluation is seen as 
an important part of the total knowledge 
base on which decisions are based. 



4. User Involvement and 
Advocacy 



b. Information users have confidence in the 
evaluation findings. 



a. 



b. 



Key personnel have a sense of ownership 
of the evaluation results, and believe 
that results will validly inform decisions" 
making • 

Information users are willing to seriously* 
consider evaluation results in the context 
of their decision-making. 



[: ERjfc 



A fifth factor, information Processing", is described to include such conditions 
as clarity of reports, and unique characteristics of information required by dif- 
ferent decisionmakers. We feel there is much overlap with "Communication" in this 
factor, end so have elected to omit Information Processing from our schema of 
Utilization structure. • 



th^fchir factor structure we haVe chosen to represent the notion of evaluation 
utiJJvMtion can be, used, therefore, to develop a measurable definition of utiliza- 
tion. We propose the following definition: 

Evaluation utilization occurs when decisions are made in the context of: 

1. Belief in the relevance of evaluation information 

2. Awareness of the importance of communicating evaluation results 
to all potential users 

3. Faith in the credibility of the data, and 

4. User involvement and advocacy of evaluation as decision- m a k ing tool. 
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Instructions for Developing Item Concepts 



The Region V TAG used a modified Nominal Group Technique (NGT) as described by 
Delbecq et al (1975) to generate items for the four columns and five rows of the 
matrix. Based upon our experience with this method of item concept generation, 
we offer the following suggestions for a procedure to be used by all TACs. 

1. Appoint one person to assume responsibility for coordinating the item con- 
cept writing effort. This person -should carefully read the position paper, 
identify persons to participate in item concept writing, distribute copies 
of paper to identified writers, chair two meetings of writers, and send in 
the resulting item concepts to Region V. 

4 

We suggest that each TAC focus on writing item concepts for two columns (to 
be decided at this TAC Directors 9 Meeting) of the. matrix. Each individual 
item writer should write items for only one column. 

2. The item writing coordinator should ^onvene a meeting to explain the model 
and the nature of the proposed instrument. This will involve: 

a. describing the four factor model derived from the Levi ton and Hughes 
paper ; 

. b. ■ describing the five Title I decision areas; 

c. displaying the matrix, and sample item cbncepts for one column (provided); 

d. describing the two scales on which LEA respondents will be asked to rate 
each item. Namely, an LEA Title . I Evaluator and/or Title I Program 
Administrator will be asked to/rate the .extent to which, the condition 
described in the item exists xn his/her district, and also to rate the 
extent to which Technical Assistance would be welcomed to facilitate 
the described condition; * 

e. assigning one column to each individual item writer. 

3. - Item concept writing may be done in the meeting, or individually. Our 

experience suggests that individual item concept writing might be most 
efficient, once all participants fully understand the task. 

4*. All items should be returned to the Coordinator who will have them typed 
and copies made for each item concept writer.' 

" \ 

5* At a second meetingV\items are discussed and ranked, by matrix cell. Dupli- 
cate items and/or related concepts may be eliminated or combined and the 
list refined prior to ranking. 

6. All item concepts, ranked in the second meeting, should be returned to 
1 Laura Crane, Region V, by November 3, 1980. 
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Matrix Column 3: EVALUATION CREDIBILITY 

DECISION AREA • ITEMS (write two for each • 

decision area) 

' i ii ■ ■ i i i i ii i 

1. Funds Allocation , ^.u r * «. 

1. Title I Evaluation Data is an important info source^in grade allocation of * 
Title I funds. 



2. -Program Adoption 
or change 

1. Title I EvaJAiation Data is usually a very accurate source in pinpointing 
program problems. 

2. -Title I Evaluation Data is normally considered in the context of program 
planning. 



3. Staffing 

1. Title I Evaluatipn Data provides an accurate assessment of staffing problems. 



4. Student Selection 

1. Student test data is a major basis (but not only basis) for project selection, 



5. Test Selection 

1. Title I tests are routinely examined for appropriateness to program goals. 

2. " Title I test result* are routinely examined for validity (Proper levels 
administered, etc*). * 9 



UTILIZATION OF EVALUATION INFORMATION" 
ITEM MATRIX 



* 1 r 

DECISION AREA 

• 


Relevance 


/ 

FACTOR 
-Communication . 


• 

Credibility | User Involvement 
1 


Funds Allocation 




» 


I 
i 

• 

i 


— K — 

Program Adoption 
or Change 


* 


1 


i * 

T - 

^ ' l- 

i 

! 


\ • 
Staffing . 




1 

1 

I 

| 


< * • 

1 

i 

* \ 

* \ 

\ 

i 
i 

i 


Student . Selection 


i 
i 

i 
1 

! 


• • 

t — , — - — 


Test Selection 

• 

• 

= ZT* 


f 

tt # 
I <v i % ■ : 
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30-Item Subset 
-Relevance 
Credibility 
Communication 
User Involvement 
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• :<: . -RELEVANCE , " 



' 5, .EVALUATION" DATA ARE USED TO DECIDE WHICH TEST BEST MATCHES THE OBJECTIVES ' 

. OF THE PROJECT. 

• • • * . • 

* » *" ■ " • 

♦ 

/7. STAFF WHO" ADMINISTER AND USE TESTS CHECK WHETHER OR NOT THE TEST CONTENT 

t 

MATCHES INSTRUCTION BEFORE SELECTING TESTS. , • • ' • 

1 

3. Student performance data are available in time to assist in student selection. 

4 * 

* # t * * * * 

24. Title I evaluation reports are received in time to assist administrators in 

MAKING DECISIONS ABOUT CHANGING THE PROJECT (s) , 



30. Evaluation data are available in time to be useful in decisions concerning 
test' selection changes'. * . i 



33. Evaluation information from the previous year is available in time to inform 

THE FOLLOWING YEAR'S FUNDS ALLOCATION DECISIONS, 



JK). Evaluation data are available when staffing decisions are being made. 




C R E D I jy L I T -Y ' 

' 1 

3/ TITLE I EVALUATION PROVIDES a sufficiently credible source of i-nformmion 
, to be considered when funding decisions are being made, 

10, .Evaluation users have confidence in, the quality of the evaluation' data used 
in making funding decisions, 

* • 

► 

* * ■ • 

* 

27, Evaluation data are considered sufficiently valid for making decisions about the 
hiring of aides, 

* 

32. Program planners believe that evaluate data accurately reflect the status 

' OF A PROJECT. f J . > • . 

58, Title I students are selected with confidence on the basis of 'evaluation data, 

63. Evaluation data are considered a sufficiently credible source of information 
for making decisions about one method of instruction vs, another, 



COMMUNICATION 

21, Teachers routinely receive evaluation information about the overall, impact 
of their Title I project. 

28, A DESCRIPTION OF DISTRICT TEST SELECTION PROCEDURES>S--AVAI LABLE TO ANYONE 

* ■» 

INTERESTED, * \ 

23, Parents understand how the student selection process, works, 

• * 

31, Parents are aware of how evaluation results are used in programmatic decisions, 

34. Teachers understand the importance of adhering to the Title I Evaluation and 
Reporting System requirements for student selection. 

35. Evaluation data are presented in a form which is useful for project planning. 

39. Meetings are conducted to make sure that student selection criteria are clearly 
defined and understood by all appropriate personnel. 

49. Title I administrators *$now-. how evaluation result's can be used to determine 
. .if a new test is needed, " 

t - - - «» * 

• - \ 

53, Evaluation information useful for funding decisions is presented in an under- 
standable fashion. j ' 



USER I N V 0 L-V E M E N T 

V * 

2. Users of test information are .periodically asked for their reactions to the 
• tests in use. • * 

♦ 

8. Project administrators are routinely involved in evaluation planning so the 
. evaluation will address their specific needs. * * i 

* 

' » 4 

15, Title I teachers have access, to decision-makers to voice their suggestions 

FOR PROJECT CHANGES. * 

* 

37. Title I staff review evaluation results for purposes of^allocating project 
funds to different project components. ' ' 

43. Persons responsible for funding decisions are routinely involved in evaluation . 
planning activities. 

57/ Project decision-makers are willing to spend the time required to insure the 
usefulness of evaluations for their needs, 

EL, Persons responsible, for making project change decisions are formally involved 

IN EVALUATION VlANNING ACTIVITIES, 
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Subsample A Correlation Matrix 
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.17607 
" . 20288" • 
.14547 
. C8627 
. 10730 

-rxrouo 

• 10 314 
.58530 
1. 00000 
. C9742 
29134 
.32 331 
.17*76 
7"l"8o55~~ 



I2E 




I13E • 

.25482 
- "."0r3572~ 
.3 56 84. 
.37211 
...1A6 88 „ 
.36044 
.148 81 
.19344 
.24 507 
TT26Z3 
Y 2 51 51 
.15197 
.35743 
.2"5357""~' 
. 21128 

-Aim 

"727543 

. 29912 
.14460 
.305 r 



I24E 

■ .12684 
— ."185T3"" 
.10714 
.43024 

^_4.0.667_.. 

* .27399 
.23538 
.34950 
.34863 

'■-V01T29-"" 
..23659 
.16992 
.32523 

" .03440™ 
.36735 
.17200 
.20123 * 



I30£ 

.12713 

■— ."28T27-- 
.21767 

.41211 

_.»1.8.250 „. 
.36777 
.20260 
. 35276 
.38391 



m- 




."1791 

• .19591 
.33941 
.89742 
1T0 OTTO 0" 
.33916 
.35931 

-rfWfl- 



.2/499 

.30974 
.31077 
.35726 
"."27990 — 

.35377 
.28468 
.2913* 
""33916— 
11.00080 
.65308 



.TI4"9W 
.22198. 
.14524 
.36162 
;"18591 
.38858 

'Mill 



I33E 



.03431 
V12T14T" 



.1 4702 
.39698 
,23696 
.2553 3 
.19415 
. 23055 
.30882 
-=TO"2172- 
.03336 
.13820 
.-32406 
"-.TJT832 
.28401 

:!SHi 



■ . 31 sm — 

.36900 
.39240 
•. 39942 ' 

"~.Tr8"422 

.30475 
.48820 
.32331 

"T3593r 

.65308* 
1.00000 



mil: — -'Mm- - u .m 



72zm~ 

.26573 
.42428 
.39421 

";^1T88" 
3 . 05917 
.2319 0 
.17475 
"."21225" 
.67215 
54382 

l.opgoo 

.348X1' 



I40E 

.25 893 
•"T24"32T 
.16975 
.46500 
. .36867 
.4795T 
.24956 
.22940 
.42542 



~7T4"84"4" 
.29354 
. 10572 
.49300 
Y24455 
i34953 

-.mil 



"734670 
.*1474 
.29716 
.26900 

"~."34'90T. 
.25 379 
.20319 
.18 665 

■ J "OTH" 
.45956 
.35952 
.3481«1 

1.00'OOflf 



FILE^- SAfEUS (CRE ATEC - 01/28/82) 

OBLIQUE F ACTOpy PATTERN MATRIX 
\ AFTER RO"TAT ION WITH KAISER; NORMAL I Z"A T IjO IT ~ 

0 

Subsample A Pattern Matrix 



DELTA = 



12E 




FACTOR 1 



00195 
13670 
05896 
.26338 
.09355 
.17036 
-.08666 



.U464U 
.15284 
.0 0571 

-«.lfl27§ 
. 079.05 
. 35983 
.19300 
-.01158 
"T9"OT3~ 

-:HM§- 

.17261 
. 09677 
6507 



_F ACTOR ,2. 

.16067 
.2 03-84 
.08424 
' .154,96 
-.15526 
• .00817 
.06178 



FACTOR 3 FACTOR V4- FACTOR 5 



!.04h20 
.05616 
.l|2.62. 

••15*o3 
.02296 

♦.0570 5 
.04465 

707377" 



-.14166 

::KiI! 

-.19323 
. 09586 
-.13114 
-. 06942 



7TJTOT 
.05721 
• 05154 
-. 03337 
-.254~58 
-'.04641 
-.13218 
-.04043 
~7C2"9T5 



.05A08 

.05781 
.01535 
-.200*0 
♦ 04A38 



-:43S53 
-.214*8 

•:»Kt 

-.2621'8 
-.13096 
06€13 
AH98 
4T2T" 



. 04135 
-. C258& 
-.136** 
.30949 
. C3256 
. 00966 
1372 



. 31*29 
-•07398 
. C4332 
-.00358 
.15023 
-. C7462 
• .11129 
~-TT3ZT6T 
. n4?e>s 



.06740 
.016U9 



« S?tt 




■-HH- 

I32E 
I58E 



T5 




-.16110 

-.16*66 



b 

9' 

.00738 
.01862 



r 

. 02680 
tV -.10208 
' -.15657 

"".0 8074" 
. 04839 
.17238 



m§ — sL tvm — a — shhss 



70*292 

:SH2 

. 30306 
.12698 

Mm 



•IPC 

I7E 

I13E 

I24E 

I30E 

I33E 

I40E 



.54798 
.11844 
.02226 
.25663 
.. 002«f7 
.04877 



. JO 

. 02112 
-.02188 
-.18183 
-709*02; 

.15306 
-. 022*3 



-rfl-8-8-94 
. 04237 
.04788 
r . 07724 
•.01509 
.08134 



,13846 
. 194 98. 
.9153.9 
".'533o9 
. 768*2 
.35905 



I2E 
I8E 
I15E 



-, T37F" 
.! I43E 





: tat 

" 12 7 E 

I32E 
I58E 



I7E 
I13E 
I24E 
T3T 




FACTOR 6 'FACTOR _7 FACT OR,. ...8_ FACTOR _9 



.15301 
.48301 
.39469 



.0T9"1T2- 
.87653 
21534 

L2j 



.05640 
.09395 
.12170 



-turn 

.06274 
-.0 40 88 
. 20026 

""TroTnnr 

.06120 
-. 02.855 
-. 04851 



T0"2X37~ 
•. 03231 
.12212 



•.34 743- 
-.12484 
.01586 



.19371 
.10276- 



-•"23T5T 
-. 05449 
■•. 0 3579 
.1J9JLDJ 



.03437 
-.19421 
.32239 



m 



TT7T5T 
.15134 
.33855 



01808 



.19829 
■;"047"4O 
. 14326 
.16108 



7l2~6Y2 
.13173 
.42054 
•09001 
.T3275" 
5T 



T3*\ . 
•. 05838 
.07339 
. 08509 
Y029W 
-.09328 
•. 08470 
•_02073 
";0"0 9"60' 



-.01832 
.01836 

-.05753 

--.UT78T 
.22702 
.19 765 
• 0.0 04J0. 

"-;D9'6~0 0 



.10671 

"iivomr 



0 0,96 9 

2. 



0 7283 
0038C 
05495 



"."06594 

11 



.0129* 
V22T66 
.10191 
-. 00562 

•roiTn - 

-.13958 
.11176 



.24335" 
.40327 
70772h" 
. 00391 
.612*4 
0?Cftq 



. 00664 
-.20T9tf 
. 02^93 
• 05743 
."02747- 
.03570 
.00692 



.10188 
"T01261.- 
. 06411 
-.0 3914 

-.nn7ft< 



♦.2072 
±i 2-479, 

.5640? 
-.05428 
"»Y11i32 
. • .07255 

.21225 



7*, 



3 . 



Subsample" A Structure Matrix 
FACTOR 1 FACT 0 R 2 F A CTO R __3 FA C TOR 4 FACTOR 5 



I2E 

S _...I15£ 
*<r I37E 




.1 3625 
. 39134 
.37237 
. 49103 
.t+5021 
.52101 

t» 

.34564 
."OS 51 7 
.201*66 
.306i*6 
. 59642 
•2S869 
,38072 



.26650 
.35752 
.19682 
.35477 

:!iKt 



'. 22072 
•. 29127 
'. 27001 
•.42383 
•.20006 
-.42531 
.2796: 



.07769 
.43602 
.36311. 
.26278 

-.-Mil 



79" 74 85 
.50^52 



..13071 
.07286 
.25538 

•'.012313 
. 21043 

-.am 

rOB85B"~ 



!76TT 
.17845 
-.09874 
-.27461 
-•."3839"5' 
-. 36061 
-.27532 
-.31835 

-T35TBY" 



?5hT 
.39515 
.13203 
.43627 
-.3970"4 
.42263 
•lt 3 37 
.99280 
.51011" 



♦50549 
.5 484~6" 
.40611 
. 33156 
.2 795 3 



am — — srftHt 

28193 . -. 38132 -.49037 



.14023 
. 15725 
.20460 
.1 8666 
.35954 
.27301 
.•+-,948 



. 28193 
T199T3" 
.10811 
• 0 2656 
.2692 



r76T6~ 
.64105 
. 23134 
.27801 
.45393 
.21230 
.14769 



-. 38132 

74256 0" 
-. 36135 
'.33522 
-.99155 



■•19fc04 
-.14766 
-.22833 
-.37177 
•". 342 82" 
-. 09366 
-. 30550 



• 4903 
733"47'S-" 
.28278 
.27278 
.26672 



77TS31T 
.27488 
.2155 7 
.34342 
V3689"2- 
•.26668 
.36560 



.1452H 
.21837 
.14621 
.51474 

-Mil 

.26 0 



T3547T 
.42620 
. C1390 
.23629 
.17003 
.41375 



T23¥5T 

ti8H 

.33310 
.373B9 
. 44211 
.23501 
.2-+231 



.3U419" 
.29954 
. 37468 
.51509 
7"c'9'972~ 
.76562 
.52423 



*4 



„ J-ACJlQS™ 6 JFAJCIJRF Z FACiOjELl FAC.LO.R L 9. 



35! 



I2E 

i I8E 
> 6 ' I15E 



a\ I37E 

1 III 



.22062 
.61007 
.52039 
•30450 
,87947 
.49051 



... I28E 
i I29E 
* I31E 
— 134 k 



.15299 
.08946 
.0 6361 
TI5367 
.17*04 
.32069 




.26755 
.12952 
.41611 

' 3 80 33 
.16748 




-.36561 
-. 10071 
. 05 027 
- . Z2T33" " 
,00690 
•00004 
147" 



"74534* 

'Mil 

.45762 
V2T72 5 — *• 
.33679 
.34411 
31 



.20675 
.8046* 
.75195 

"724948" 
.29843 
•48081 
.12002. 

V29244 



-. 017*7 
. 00555 
•02900 

'939" 



•05006 
i «"D3"8W 
.14146 
.25158 



I24E 



.24263 
.17954' 
.15180 



-. 06083 
-. 09942 
.11425 

.T371T3"- 

=t» 

.*3024 
"T11-75Z- 
. 04426 




.20152 
.34435 



.29603 
. 211 0 0 
.09231 



ft 



.11834 
•• 0 0 93.0 
.4126 8 

"73977F" 
.30962 
.51776 

.15346 
.10865 
.11087 
-T1T24-5' 
•44534 
.31196 
.14909 
i 2152 7 

■rfKH 

.28873 
■'.T4-48-8- 
.26 05 7 
.20 601 



-.05452 
i 24934 



-.ilj 

-•27423 
.01871 
.08867 



333 



'.0207F 
.07982 
. 04478 



49- 



-. 09773 
.62031 
.17224 



T3T8~4~6" 
.24266 
.40660 



Subsaople A Factor Correlations 



FACTOR '1 FACTOR 2 FACTOR 3 FACTOR 4 FACTOR 5 



1.00000 
.1*801 
-.36633 
-.3¥874 
.320 00 
.1*2161 
.16271 
.05199 
.32374 



.14801 

i:lim 

-^,21247 
.28732 
.19154 
.07606 

-.13004" 
. 09617 



-. 3A633 
-.16173 
1.00000 
V3Ur7«'" 
-. 28196 
-.28952 
—.20330 
'-. 02048 
-.23869 



-.39871* 
-.2124 7 
.30174 
T.T0WD 
-. 35216 
-.35843 
-.20996 
-YD 9 4 20 
-.15674 



.32000 
.§873? 

-.Z819o 
"-.'152X6- 

1.00000 
.22 706 t 
.10,789 
. C4 017 ' 
.25615 



FACTOR f 

.42161 
.19154 
• -.28952 
-.35843 
" .22706" 
1.00000 
.22393 
04104 
" .18878 



FACTOR 7 FACTOR 8 FACTOR 9 



.16271 
.0 760 6 
.20330 
.20996 

.22393 
.00000 
.0654_9 
; 14 733" 



.05199 
-.13004 
-.02048 
-.09^20 

—.uwinrr 

. 04104 
-.06549 
"1.J0 00 0 

. 0"l55 0 



.32374 
. 09617 
-.23869 
-.15674 



.25fclfa 
.18878 
.14733 
_. 0155 0 

"1.6'OBoT 




